Method to expedite playing of binaural sound to a listener

ABSTRACT

A method expedites processing and playing of binaural sound during an electronic communication between a first user and a second user. An electronic device of the first user convolves sound into binaural sound for the second user before the binaural sound transmits to the electronic device of the second user. In this way, the binaural sound is already convolved and ready to play upon receipt at the electronic device of the second user.

BACKGROUND

Three-dimensional (3D) sound localization offers people a wealth of newtechnological avenues to not merely communicate with each other but alsoto communicate with electronic devices, software programs, andprocesses.

As this technology develops, challenges will arise with regard to howsound localization integrates into the modern era. Example embodimentsoffer solutions to some of these challenges and assist in providingtechnological advancements in methods and apparatus using 3D soundlocalization.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a method that expedites processing and/or playing of binauralsound during an electronic communication between a first user and asecond user in accordance with an example embodiment.

FIG. 2 is a method that allows or denies a user access to HRTFs of otherusers in accordance with an example embodiment.

FIG. 3 is a method that allows or denies a first user to providebinaural sound to a second user in accordance with an exampleembodiment.

FIG. 4 is an electronic device with a display displaying privacysettings that enable a user to control access to HRTFs of the user inaccordance with an example embodiment.

FIG. 5 is an electronic device with a display displaying privacysettings that enable a user to control playing of binaural sound or 3Dsound to the user in accordance with an example embodiment.

FIG. 6 is an example computer system in accordance with an exampleembodiment.

FIG. 7 is an example of an electronic device in accordance with anexample embodiment.

SUMMARY

Example embodiments include methods and apparatus that expediteprocessing and/or playing of binaural sound to a listener.

During an electronic communication between a first user and a seconduser, an electronic device of the first user convolves sound intobinaural sound for the second user before the binaural sound transmitsto the electronic device of the second user. In this way, processing orplaying of binaural sound is expedited since the binaural sound isalready convolved and ready to play upon receipt at the electronicdevice of the second user.

Example embodiments include methods and apparatus that enable users tocontrol access to HRTFs and playing of binaural sound to the users.

Other example embodiments are discussed herein.

DETAILED DESCRIPTION

Binaural sound or three-dimensional (3D) sound externally localizes awayfrom a head of the listener, unlike stereo or mono sound that localizesinside the head of the listener or localizes to a physical soundspeaker. Thus, when a listener hears binaural sound, a source orlocation of the sound occurs outside the head of the listener eventhough this location may be in empty space or space not occupied with aphysical sound speaker.

Binaural sound has many technical challenges and problems, especiallywhen users exchange or play binaural sound during an electroniccommunication. Example embodiments offer solutions to these challengesand problems.

One problem during an electronic communication is that processing orconvolving sound with head-related transfer functions (HRTFs) is processintensive. Electronic devices often use a dedicated or specializedprocessor, such as a digital signal processor (DSP), to perform the taskof convolving sound into binaural sound. This task can be especiallyprocess intensive if the head of the listener is moving, the sound ismoving with respect to the listener, or the sound must be repeatedlyconvolved with different pairs of HRTFs. If the processor in theelectronic device cannot convolve the sound quickly enough, the soundthe listener hears may appear delayed or jumpy and ultimately diminishthe experience of hearing binaural sound.

Example embodiments solve these problems and others.

During an electronic communication between a first user and a seconduser, an electronic device of the first user convolves sound intobinaural sound for the second user before the binaural sound transmitsto the electronic device of the second user. In this way, the binauralsound is already convolved and ready to play upon receipt at theelectronic device of the second user. This process expedites playing ofthe binaural sound at the electronic device of the second user sincethis electronic device is not required to convolve the sound intobinaural sound before the listener hears the sound. The binaural soundreceived is ready for immediate play at the electronic device of thesecond user. This process also saves processing resources at theelectronic device of the second user since a DSP at the electronicdevice of the second user is not required to convolve the sound.

Consider an example in which two users engage in an electroniccommunication, such as a telephone call or sending graphicalrepresentations that play binaural sound. The electronic device of thesecond user may not have a DSP or a capability to convolve sound.Alternatively, this electronic device may have a DSP but the DSP may bebusy or not able to convolve the sound in real-time to provide thelistener with an effective listening experience. An example embodimentsolves these problems since the electronic device of the first userconvolves the sound before sending the sound to the electronic device ofthe second user. Alternatively, a server in communication with the firstelectronic device convolves the sound. In this way, the electronicdevice of the second user is relieved of process intensive convolutiontasks and can more quickly provide the binaural sound to the seconduser.

The processing tasks of convolving binaural sound are thus provided bythe first electronic device or server instead of conventional techniquesin which such processing tasks of convolving the sound are provided bythe second electronic device.

Another problem is that users may not want to share their customizedHRTFs with other people or have unauthorized people, electronic devices,or other entities access their customized HRTFs. If a third partyobtains the customized HRTFs of a user, then this third party cancontrol where binaural sound originates to the user. The third party canconvolve sounds so they appear outside and away from the head of theuser even though the user does not want to hear the sound at theselocations. The parties can thus invade the privacy or listening space ofthe user.

Example embodiments restrict who or what can access customized HRTFs ofa user. If a person, electronic device, and/or software program does nothave authorization to access the customized HRTFs of a user, then accessis denied. By way of example, users can control access through privacysettings or other example embodiments discussed herein.

Another problem is that users may not want to hear sound as binauralsound or may want to restrict who can send and/or play binaural sound tothem. For example, a user might not want to give advertisers orstrangers the ability to send and/or to play binaural sound. Users maywant to restrict what people, electronic devices, or software programshave the ability to send and/or play binaural sound to the users.

Example embodiments solve these problems and others.

Example embodiments restrict who or what can send and/or play binauralsound to a user. If a person, electronic device, and/or software programdoes not have authorization to send and/or play binaural sound to auser, then playing of the binaural sound is rejected or prohibited. Thesound may not play to the user, or the sound may be converted to monosound or stereo sound and then played to the user. By way of example,users can control the sending and/or playing of binaural sound throughprivacy settings or other example embodiments discussed herein.

Consider an example in which an advertiser telephones a user. Theadvertiser has HRTFs for the user and convolves the sound with theseHRTFs. When the user answers the call, the user hears a computer programvoice of an advertisement. Since the user is wearing headphones orearphones, this voice externally localizes as binaural sound about onemeter away from the head of the user. The user, however, did not want tohear the voice as binaural sound since a friend of the user was sittingat the location where the voice externally localized. Exampleembodiments solve this problem since the user is able to determine inadvance who can play binaural sound to the user. In this instance, theuser could have set privacy settings to reject the incoming call asbinaural sound and instead play the incoming call as mono sound orstereo sound.

FIG. 1 is a method that expedites processing and/or playing of binauralsound during an electronic communication between a first user and asecond user in accordance with an example embodiment.

Block 100 states obtain, for an electronic communication between a firstuser with a first electronic device and a second user with a secondelectronic device, sound with the first electronic device to play to thesecond user at the second electronic device.

By way of example, a computer or electronic device generates the sound(computer-generated sound), or microphones capture and record the soundto be sent. For example, one or more microphones capture the sound asmono sound or stereo sound when the first user speaks a message or avoice call to the second user. As another example, the first electronicdevice or a server in communication with the first electronic deviceincludes a plurality pre-recorded or previously generated sounds thatwill play to the second user.

The sound can be stored in memory of an electronic device, obtained frommemory of an electronic device (such as a computer or server), and/ortransmitted or streamed over one or more networks.

Consider an example in which the first electronic device executes amobile software messaging application that includes hundreds orthousands of sound clips or sound files. The first electronic deviceobtains or has access to these sound clips or sound files and can sendthem to other users of the mobile software messaging application.

Consider an example in which the first electronic device obtains thesound when the first user speaks into microphones in the firstelectronic device or in communication with the first electronic device.The microphones records the voice of the first user as he or she recordsa message or sound to be played to the second user.

Consider an example in which the first and second users talk to eachother during an electronic call, telephony call, or telephone call(e.g., a Voice over Internet Protocol or VoIP call). One or moremicrophones in or in communication with the electronic device of thefirst user capture the voice of the first user.

Block 110 states obtain, with the first electronic device or a server incommunication with the first electronic device, head-related transferfunctions (HRTFs) of the second user.

For example, the electronic device retrieves or receives the HRTFs ofthe second user from memory, such retrieving them from the firstelectronic device, from a server, from a database, from a networklocation, etc.

The HRTFs can be generic HRTFs, customized HRTFs, or HRTFs that arecustomized to the listener. Customized HRTFs or HRTFs that arecustomized to the listener are specific to an anatomy of a particularlistener and are based on a size and/or shape of the head and/or ears ofthe listener. Customized HRTFs can be obtained from actual measurements(e.g., measuring HRIRs and/or BRIRs from a head of the user) or fromcomputational modeling (e.g., modeled from a photo of the user ormodeled from measurements or approximations of the listener, such as asize and/or shape of the listener's head or ears). Customized HRTFs arealso known as individualized HRTFs.

Generic HRTFs are not specific to an anatomy of the listener. GenericHRTFs can be obtained from actual measurements (e.g., measuring HRIRsand/or BRIRs from a head of the user or a dummy head) or fromcomputation modeling. Generic HRTFs can work for a large group of peoplesince these HRTFs are not customized or individualized to each person.These HRTFs are often stored in public databases and available to thegenerally public to use free of charge.

Although both generic and customized HRTFs can be used to convolvesound, customized HRTFs work better in that they more accurately andconsistently externally localize sound to the listener. This is truebecause customized HRTFs enable the listener to hear sound providedthrough speakers in a same way as hearing sound in their environment.Customized HRTFs copy or emulate the impulse responses of the human headand ears of the listener.

Generic HRTFs can successfully convolve sound to external locations forsome people but not others. Further, different users hearing soundconvolved with generic HRTFs may hear the sound localizing fromdifferent locations. For instance, user 1 hears the sound externallylocalizing in empty space one meter away and on a right side of his orher head. User 2 hears the same sound externally localizing in emptyspace behind his or her head. User 3 hears the same sound with noexternal localization but instead hears the sound originating inside hisor her head. By contrast, if the sound were convolved with customizedHRTFs particular for each user, user 1, 2, and 3 will hear the soundexternally localizing to the same relative location with respect totheir head.

Block 120 states process and/or convolve, with the first electronicdevice or the server in communication with the first electronic device,the sound with the HRTFs of the second user.

For example, a processor (such as a DSP) processes or convolves thesound with one or more of head-related transfer functions (HRTFs),head-related impulse responses (HRIRs), room impulse responses (RIRs),room transfer functions (RTFs), binaural room impulse responses (BRIRs),binaural room transfer functions (BRTFS), interaural time delays (ITDs),interaural level differences (ITDs), and a sound impulse response.

An example embodiment processes or convolves the sound with the HRTFsbefore the sound is provided to the electronic device of the seconduser. This expedites processing and/or playing of the sound to thesecond user since the second user does not have to wait while aprocessor processes or convolves the sound into binaural sound.Furthermore, the processor (such as the DSP) in the electronic device ofthe second user can be devoted to other tasks instead of convolving thesound into binaural sound. Furthermore, the electronic device of thesecond user may not have a dedicated DSP or processor capable ofconvolving sound.

Sound includes, but is not limited to, one or more of stereo sound, monosound, binaural sound, computer-generated sound, sound captured withmicrophones, and other sound. Furthermore, sound includes differenttypes including, but not limited to, music, background sound orbackground noise, human voice, computer-generated voice, and othernaturally occurring or computer-generated sound.

When the sound is recorded or generated in mono sound or stereo sound,convolution changes the sound to binaural sound. For example, one ormore microphones record a human person speaking in mono sound or stereosound, and a processor processes this sound with filters to change thesound into binaural sound.

The processor or sound hardware processing or convolving the sound canbe located in one or more electronic devices or computers including, butnot limited to, headphones, smartphones, tablet computers, electronicspeakers, head mounted displays (HMDs), optical head mounted displays(OHMDs), electronic glasses (e.g., glasses that provide augmentedreality (AR)), servers, portable electronic devices (PEDs), handheldportable electronic devices (HPEDs), wearable electronic devices (WEDs),and other portable and non-portable electronic devices. These electronicdevices can also be used to execute example embodiments.

In one example embodiment, the DSP is located in the electronic deviceof the first user. In other example embodiments, the DSP is located inother electronic devices, such as a server in communication with thefirst and second electronic devices.

The DSP processes or convolves stereo sound or mono sound with a processknown as binaural synthesis or binaural processing to provide the soundwith sound localization cues (ILD, ITD, and/or HRTFs) so the listenerexternally localizes the sound as binaural sound or 3D sound.

An example embodiment models the HRTFs with one or more filters, such asa digital filter, a finite impulse response (FIR) filter, an infiniteimpulse response (IIR) filter, etc. Further, an ITD can be modeled as aseparate delay line.

When the binaural sound is not captured (e.g., on a dummy head or humanhead), the captured sound is convolved with sound localizationinformation (SLI). This information includes one or more of HRTFs,HRIRs, BRTFs, BRIRs, ILDs, ITDs, and/or other information discussedherein. By way of example, SLI are retrieved, obtained, or received frommemory, a database, a file, an electronic device (such as a server,cloud-based storage, or another electronic device in the computer systemor in communication with a PED providing the sound to the user throughone or more networks), etc. Instead of being retrieved from memory, thisinformation can also be calculated in real-time.

A central processing unit (CPU), processor (such as a DSP), ormicroprocessor processes and/or convolves the sound with the SLI, suchas a pair of head related transfer functions (HRTFs), ITDs, and/or ILDsso that the sound will localize to a zone, area, or sound localizationpoint (SLP). For example, the sound localizes to a specific point (e.g.,localizing to point (r, θ, ϕ)) or a general location or area (e.g.,localizing to far-field location (θ, ϕ) or near-field location (θ, ϕ)).As an example, a lookup table that stores a set of HRTF pairs includes afield/column that specifies the coordinates associated with each pair,and the coordinates indicate the location for the origination of thesound. These coordinates include a distance (r) or near-field orfar-field designation, an azimuth angle (θ), and/or an elevation angle(ϕ).

The complex and unique shape of the human pinnae transforms sound wavesthrough spectral modifications as the sound waves enter the ear. Thesespectral modifications are a function of the position of the source ofsound with respect to the ears along with the physical shape of thepinnae that together cause a unique set of modifications to the soundcalled head related transfer functions or HRTFs.

A unique pair of HRTFs (one for the left ear and one for the right ear)can be modeled or measured for each position of the source of sound withrespect to a listener as the customized HRTFs.

A HRTF is a function of frequency (f) and three spatial variables, byway of example (r, θ, ϕ) in a spherical coordinate system. Here, r isthe radial distance from a recording point where the sound is recordedor a distance from a listening point where the sound is heard to anorigination or generation point of the sound; θ (theta) is the azimuthangle between a forward-facing user at the recording or listening pointand the direction of the origination or generation point of the soundrelative to the user; and ϕ (phi) is the polar angle, elevation, orelevation angle between a forward-facing user at the recording orlistening point and the direction of the origination or generation pointof the sound relative to the user. By way of example, the value of (r)can be a distance (such as a numeric value) from an origin of sound to arecording point (e.g., when the sound is recorded with microphones) or adistance from a SLP to a head of a listener (e.g., when the sound isgenerated with a computer program or otherwise provided to a listener).

When the distance (r) is greater than or equal to about one meter (1 m)as measured from the capture point (e.g., the head of the person) to theorigination point of a sound, the sound attenuates inversely with thedistance. One meter or thereabout defines a practical boundary betweennear-field and far-field distances and corresponding HRTFs. A“near-field” distance is one measured at about one meter or less;whereas a “far-field” distance is one measured at about one meter ormore. Example embodiments are implemented with near-field and far-fielddistances.

The coordinates for external sound localization can be calculated orestimated from an interaural time difference (ITD) of the sound betweentwo ears. ITD is related to the azimuth angle according to, for example,the Woodworth model that provides a frequency independent ray tracingmethodology. The coordinates (r, θ, ϕ) for external sound localizationcan also be calculated from a measurement of an orientation of and adistance to the face of the person when a head related impulse response(HRIR) is captured.

The coordinates can also be calculated or extracted from one or moreHRTF data files, for example by parsing known HRTF file formats, and/orHRTF file information. For example, HRTF data is stored as a set ofangles that are provided in a file or header of a file (or in anotherpredetermined or known location of a file or computer readable medium).The data can include one or more of time domain impulse responses (FIRfilter coefficients), filter feedback coefficients, and an ITD value.This information can also be referred to as “a” and “b” coefficients. Byway of example, these coefficients are stored or ordered according tolowest azimuth to highest azimuth for different elevation angles. TheHRTF file can also include other information, such as the sampling rate,the number of elevation angles, the number of HRTFs stored, ITDs, a listof the elevation and azimuth angles, a unique identification for theHRTF pair, and other information. The data can be arranged according toone or more standard or proprietary file formats, such as AES69, andextracted from the file.

The coordinates and other HRTF information are calculated or extractedfrom the HRTF data files. A unique set of HRTF information (including r,θ, ϕ) is determined for each unique HRTF.

The coordinates and other HRTF information are also stored in andretrieved from memory, such as storing the information in a look-uptable. The information is quickly retrieved to enable real-timeprocessing and convolving of sound using HRTFs and hence improvescomputer performance of execution of binaural sound.

The SLP represents a location where a person will perceive an origin ofthe sound. For an external localization, the SLP is away from the person(e.g., the SLP is away from but proximate to the person or away from butnot proximate to the person). The SLP can also be located inside thehead of the person (e.g., when the sound is provided as mono sound orstereo sound). Sound can also switch between externally localizing andinternally localizing, such as appearing to move and pass through a headof a listener.

SLI can also be approximated or interpolated based on known data orknown SLI, such as SLI for other coordinate locations. For example, aSLP is desired to localize at coordinate location (2.0 m, 0°, 40°), butHRTFs for the location are not known. HRTFs are known for twoneighboring locations, such as known for (2.0 m, 0°, 35°) and (2.0 m,0°, 45°), and the HRTFs for the desired location of (2.0 m, 0°, 40°) areapproximated from the two known locations. These approximated HRTFs areprovided to convolve sound to localize at the desired coordinatelocation (2.0 m, 0°, 40°).

Sound is convolved either directly in the time domain with a finiteimpulse response (FIR) filter or with a Fast Fourier Transform (FFT).For example, an electronic device convolves the sound to one or moreSLPs using a set of HRTFs, HRIRs, BRIRs, or RIRs and provides the personwith binaural sound.

In an example embodiment, convolution involves an audio input signal andone or more impulse responses of a sound originating from variouspositions with respect to the listener. The input signal is a limitedlength audio signal (such as a pre-recorded digital audio file or soundclip) or an ongoing audio signal (such as sound from a microphone orstreaming audio over the Internet from a continuous source). The impulseresponses are a set of HRIRs, BRIRs, RIRs, etc.

Convolution applies one or more FIR filters to the input signals andconvolves the input signals into binaural audio output or binauralstereo tracks. For example, the input signals are convolved intobinaural audio output that is specific or individualized for thelistener based on one or more of the impulse responses to the listener.

The FIR filters are derived binaural impulse responses. Alternatively oradditionally, the FIR filters are obtained from another source, such asgenerated from a computer simulation or estimation, generated from adummy head, retrieved from storage, computed based on known impulseresponses captured from people, etc. Further, convolution of an inputsignal into binaural output can include sound with one or more ofreverberation, single echoes, frequency coloring, and spatialimpression.

Processing of the sound also includes calculating and/or adjusting aninteraural time difference (ITD), an interaural level difference (ILD),and/or other aspects of the sound in order to alter the cues andartificially alter the point of localization. Consider an example inwhich the ITD is calculated for a location (θ, ϕ) with discrete Fouriertransforms (DFTs) calculated for the left and right ears. The ITD islocated at the point for which the function attains its maximum value,known as the argument of the maximum or arg max as follows:

${ITD} = {\arg\;{\max(\tau)}{\sum\limits_{n}{{d_{l,\theta,\phi}(n)} \cdot {{d_{r,\theta,\phi}( {n + \tau} )}.}}}}$

Subsequent sounds are filtered with the left HRTF, right HRTF, and/orITD so that the sound localizes at (r, θ, ϕ). Such sounds includefiltering stereo and monaural sound to localize at (r, θ, ϕ). Forexample, given an input signal as a monaural sound signal s(n), thissound is convolved to appear at (θ, ϕ) when the left ear is presentedwith:s _(l)(n)=s(n−ITD)·d _(l,θ,ϕ)(n);and the right ear is presented with:s _(r)(n)=s(n)·d _(r,θ,ϕ)(n).

Consider an example in which a dedicated digital signal processor (DSP)executes frequency domain processing to generate real-time convolutionof monophonic sound to binaural sound.

By way of example, a continuous audio input signal x(t) is convolvedwith a linear filter of an impulse response h(t) to generate an outputsignal y(t) as follows:

y(τ) = x(τ) ⋅ h(τ) = ∫₀^(∞)x(τ − t) ⋅ h(t) ⋅ dt.

This reduces to a summation when the impulse response has a given lengthN and the input signal and the impulse response are sampled at t=iDt asfollows:

${y(i)} = {\sum\limits_{j = 0}^{N - 1}{{x( {i - j} )} \cdot {{h(j)}.}}}$

Execution time of convolution further reduces with a Fast FourierTransform (FFT) algorithm and/or Inverse Fast Fourier Transform (IFFT)algorithm.

Consider another example of binaural synthesis in which recorded orsynthesized sound is filtered with a binaural impulse response (e.g.,HRIR or BRIR) to generate a binaural output sound to the person. Theinput sound is preprocessed to generate left and right audio streamsthat are mapped to one or more sound sources or sound localizationpoints (known as SLPs). These streams are convolved with a binauralimpulse response for the left ear and the right ear to generate the leftand right binaural output sound signal. The output sound signal isfurther processed depending on a final destination. For example, across-talk cancellation algorithm is applied to the output sound signalwhen it will be provided through loudspeakers or applying artificialbinaural reverberation to provide 3D spatial context to the sound.

Block 130 states transmit, from the first electronic device or theserver in communication with the first electronic device, the soundprocessed and/or convolved with the HRTFs to the second electronicdevice of the second user.

The first electronic device or the server transmits the sound over oneor more wired or wireless networks (e.g., a cellular network, theinternet, etc.) to the second electronic device. For example, the firstelectronic device includes a wireless transmitter/receiver that sendsthe sound.

Consider an example in which the first user commands or instructs asound clip to play to the second user during an electronic communicationbetween the first and second users. In response to this command orinstruction, the first electronic device convolves the sound withgeneric or customized HRTFs of the second user and transmits theconvolved sound clip and a 3D moving emoji to the second electronicdevice.

In another example embodiment, a server or another electronic devicetransmits the sound to the second electronic device. Consider an examplein which the first and second users talk or message each other with amobile messaging software application. The application executes on theelectronic devices and one or more servers. When the first user clickson a 3D sound emoji, this action causes one of the servers to convolvethe sound with generic or customized HRTFs and then transmit the 3Demoji and convolved sound to the second electronic device.

Block 140 states receive, from the first electronic device or the serverin communication with the first electronic device and at the secondelectronic device, the sound processed and/or convolved with the HRTFs.

The second electronic device receives the sound from the firstelectronic device or another electronic device (e.g., a server) incommunication with the first electronic. For example, the secondelectronic device includes a wireless transmitter/receiver that receivesthe sound over one or more networks.

Block 150 states play, to the second user with the second electronicdevice, the sound processed and/or convolved with the HRTFs by the firstelectronic device or the server in communication with the firstelectronic device such that the sound externally localizes as binauralsound away from a head of the second user.

The sound plays to the listener as binaural sound that externallylocalizes away from or outside of the head of the listener. For example,headphones or earphones provide this sound at one or more soundlocalization points (SLPs) discussed herein.

Consider an example in which the second user receives the binaural soundto play, such as a recorded voice message, a sound clip, voice during atelephone call, or streaming audio from the first user. When the seconduser hears this sound with headphones or earphones, the sound externallylocalizes as binaural sound outside of and away from the head of thesecond user. This SLP can be, for example, a location in empty spacewhere no tangible object exists, a location in empty space where animage exists, a location in occupied space where no electronic deviceexists (e.g., sound localizing to a stuffed animal, chair, or wall), ora location in occupied space where an electronic device exists (e.g.,sound localizing to an electronic watch with no speakers).

Consider an example in which two users talk during a telephone call. Theelectronic device of the second user, however, does not have DSP. Thevoice of the first user transmits through a server that convolves thevoice with customized HRTFs of the second user. When the electronicdevice of the second user receives the sound (here, the voice), it isalready convolved and ready for immediate play to the second user who iswearing headphones.

FIG. 2 is a method that allows or denies a user access to HRTFs of otherusers in accordance with an example embodiment.

Block 200 states store HRTFs of users in memory.

For example, generic or customized HRTFs of users are stored in anelectronic device including, but not limited to, a portable electronicdevice, a handheld portable electronic device, a hard disk, flashmemory, portable memory, a laptop computer, a tablet computer, a server,a database (e.g., a relational or graph database), a data center, or alocation on a network or cloud.

The HRTFs and/or names or identities of the users to whom the HRTFsbelong can be encrypted to protect against theft and unwanted use.

Block 210 makes a determination as to whether to allow a user to haveaccess to HRTFs of other users.

Consider an example in which users can access HRTFs of other users withaccess privileges or credentials provided to an electronic device, suchas a server, database, or HPED. By way of example, in order to accessthe HRTFs, the user or electronic device provides a username andpassword, passphrase, login ID, passcode, a key or an encryption key,facial recognition, thumb print ID, an address (e.g., an IP address orMAC address), pin code, etc. This information can also be used to unlockor decrypt the HRTFs (e.g., when they are locked or encrypted).

Consider an example in which the names or identifications of theentities having access or not having access are stored in a table.Entities include, but are not limited to, people, electronic devices,organizations, companies, friends, family, websites, software programs,etc. When an entity requests access to the HRTFs of the user, anelectronic device or software program checks the table to determinewhether the entity has access privileges or does not have accessprivileges to the HRTFs of the user. For example, if the entity is notlisted in the table as having access privileges, access is denied.Alternatively, if the entity is not listed in the table as having accessprivileges, then access is denied pending authorization from the user.As another example, if the entity did not provide the correct usernameand password or other credentials, then access is denied.

Generally, each user wants to control who or what has access to theHRTFs of the user. This enables each user to determine what entities canaccess the HRTFs of the user, especially customized HRTFs since theseare unique to each user. Users can input or provide this informationabout which entities have or do not have access to the HRTFs. Thisinformation can also be input or provided with an electronic device orsoftware program. For example, a software program automatically gathersand inputs or updates this information.

Consider an example embodiment in which a user provides or inputs namesor identifications of each entity entitled to access the HRTFs of theuser. The user can also provide names or identifications of each entitynot entitled to access the HRTFs of the user. This information is storedin memory, such as being stored in a database, file, or lookup table.

Consider an example in which a software program or an electronic deviceprovides the names or identifications of the entities that have accessprivileges or rights to the HRTFs and/or entities that do not haveaccess privileges or rights to the HRTFs. For example, the softwareprograms extracts, retrieves, gathers, receives, or obtains names fromand allows access to HRTFs for people in an electronic address book ofthe user, for people who are friends of the user at a social networkingwebsite, for people who are relatives of the user, for people who arebusiness associates or colleagues of the user, for people who execute asame software application as the user (e.g., both the user and thepeople use the same mobile messaging software application and appear asfriends or contacts to the user), for people who the user liked viasocial media, for people who are followers of the user of aninternet-based photo-sharing software application, for people who arelisted as a contact in the user's email or contacts list, for people whowork at a same company or organization as the user, for people whobelong to a same group, etc.

If the answer to the determination in block 210 is “no” then flowproceeds to block 220 that states deny the user access to the HRTFs ofthe other users.

The user or entity requesting access to the HRTFs does not haveprivileges, authentication, credentials, or permissions to access theHRTFs. For example, the user or entity did not provide a correct accesscode or password. As another example, the user was not attempting accessfrom a mobile messaging software application that is required to accessthe HRTFs of other users also executing and communicating via the mobilemessaging software application.

Consider an example in which two users (a first user and a second user)electronically communicate via a mobile messaging software applicationthat provides voice messaging, calls, texting, and sending graphicalrepresentations with sound (such as emoji and animoji). When theelectronic communication commences, the electronic device of the firstuser sends the electronic device of the second user a request forcustomized HRTFs of the second user or access to these HRTFs. The secondelectronic device consults privacy settings of the second user. Theseprivacy settings include a list of entities entitled to have access tothe customized HRTFs of the second user. The first user is not on thislist, so the second electronic device responds to the request with adenial. The first and second users are still able to electronicallycommunicate via the software application but the electronic device ofthe first user is not able to convolve sound with the customized HRTFsbefore sending sound to the electronic device of the second user.

If the answer to the determination in block 210 is “yes” then flowproceeds to block 230 that states allow the user access to the HRTFs ofthe other users.

The user or entity requesting access to the HRTFs does have privileges,authentication, credentials, or permissions to access the HRTFs. Forexample, the user or entity provides a correct access code or password.As another example, the user attempting access from a mobile messagingsoftware application is allowed access since this user appears as friendor contact with the user from whom the HRTFs are being requested.

Consider an example in which two users (a first user and a second user)electronically communicate via a mobile messaging software applicationthat provides voice messaging, calls, texting, and sending graphicalrepresentations with sound (such as emoji and animoji). When theelectronic communication commences, the electronic device of the firstuser sends the electronic device of the second user a request forcustomized HRTFs of the second user or access to these HRTFs. Theelectronic device of the second user displays a message to the seconduser with this request. For example, the requests displays as follows:“Allow access to HRTFs?” The second user affirms the request. Inresponse to this confirmation, the electronic device of the first usersends the electronic device of the second user a time-based one-timepassword (TOTP) that is computed from a shared secret and current time.The TOTP is a hash-based message authentication code (HMAC) with atimestamp that expires after a predetermined amount of time (e.g.,expire after one hour or another time). Using this TOTP, the electronicdevice of the first user downloads the customized HRTFs of the seconduser. The electronic device of the first user then convolves sound withthe customized HRTFs before sending the sound to the electronic deviceof the second user.

FIG. 3 is a method that allows or denies a first user to providebinaural sound to a second user in accordance with an exampleembodiment.

Block 300 states receive, from an electronic device of a first user andat an electronic device of a second user, an electronic communicationwith sound in binaural sound.

The electronic device of the second user receives sound in binauralsound from the first user. For example, this sound was previouslyconvolved into binaural sound or captured as binaural sound. Forexample, the first and second users exchange voice messages or graphicalrepresentations with sound in binaural sound. As another example, thefirst and second users engage in a telephone call in which the voice ofthe first user is in binaural sound.

Consider an example in which the binaural sound is convolved withgeneric HRTFs or customized HRTFs of the second user. This sound willexternally localize outside of and away from the head of the second usersince the second user wears headphones or earphones.

Block 310 makes a determination as to whether to allow the sound to playas binaural sound (aka, 3D sound).

In some instances, the second user or the listener may not want to hearsound play as binaural sound but would prefer to hear the sound play asmono sound or stereo sound. For example, the listener is located in aconfined space, and the sound would localize to an unintended objectnear the listener. As another example, several people are standing orsitting near the listener, and the sound would localize to one of thesepeople and confuse the listener as to who is talking. As anotherexample, the first user is an advertising company, manufacturer, orretailer, and the listener does not want the sound to play in his or herprivate space which the listener considers to be within a radius of twometers from his or her head. One skilled in the art can appreciate thatthere are many reasons why the second user or an electronic device ofthe second user would not want the sound to play as binaural sound butwould prefer mono or stereo sound.

Consider an example in which the second user navigates to his or heruser preferences and selects “no binaural sound” or “no 3D sound.”Subsequently, sound is played to the second user as mono or stereo soundand not binaural sound.

Consider an example in which the first user must have an accessprivilege, a permission, credentials, right, or a confirmation in orderto play binaural sound to the second user. By way of example, in orderto play binaural sound to the second user, the first user or theelectronic device of the first user must provide a username andpassword, passphrase, login ID, passcode, a key or an encryption key,facial recognition, thumb print ID, an address (e.g., an IP address orMAC address), pin code, etc.

Consider an example in which the names or identifications of theentities having privileges or rights to play binaural sound to thesecond user are stored in a table. When an entity sends binaural soundto the second user or electronic device of the second user, theelectronic device of the second user or software program checks thetable to determine whether the entity has the right to play sound to thesecond user as binaural sound. For example, if the entity is not listedin the table as having such right or privilege, playing of binauralsound is denied. Alternatively, if the entity is not listed in the tableas having such right or privilege, then access is denied pendingauthorization from the second user.

Consider an example in which electronic devices of users transmit soundover one or more wireless networks in network packets or files. In orderto play binaural sound to the second user, the network packet or filemust contain a code or key to show the first user is authorized to playthe binaural sound to the second user. The electronic device of thesecond user retrieves this code or key to determine whether a matchexists that authorizes playing of binaural sound.

Generally, each user wants to control who or what has the right to playbinaural sound. This enables each user to determine what entities canhave rights to play binaural sound and what entities do not have suchrights. Users can input or provide this information about which entitieshave or do not have such rights. This information can also be input orprovided with an electronic device or software program. For example, asoftware program automatically gathers and inputs or updates thisinformation.

Consider an example embodiment in which a user provides or inputs namesor identifications of each entity entitled to play binaural sound to theuser. The user can also provide names or identifications of each entitynot entitled to play binaural sound to the user. This information isstored in memory, such as being stored in a database, file, or lookuptable.

Consider an example in which a software program or an electronic deviceprovides the names or identifications of the entities that have accessprivileges or rights to play binaural sound and/or entities that do nothave access privileges or rights to play binaural sound. For example,the software programs extracts, retrieves, gathers, receives, or obtainsnames from people in an electronic address book of the user, for peoplewho are friends of the user at a social networking website, for peoplewho are relatives of the user, for people who are business associates orcolleagues of the user, for people who execute a same softwareapplication as the user (e.g., both the user and the people use the samemobile messaging software application and appear as friends or contactsto the user), for people who the user liked via social media, for peoplewho are followers of the user of an internet-based photo-sharingsoftware application, for people who are listed as a contact in theuser's email or contacts list, for people who work at a same company ororganization as the user, for people who belong to a same group, etc.

If the answer to the determination in block 310 is “no” then flowproceeds to block 320 that states reject the sound or play the sound tothe second user in one of mono sound or stereo sound instead of thebinaural sound.

When authorization is denied, the electronic device of the second usercan refuse to play the sound to the second user. For example, anincoming telephone call is rejected or an incoming emoji with sound isrejected.

When authorization is denied, the electronic device of the second usercan play the sound to the second user as one of mono sound or stereosound instead of the binaural sound. For example, an incoming telephonecall or incoming emoji with sound is accepted but the sound plays to thesecond user as mono or stereo sound and not binaural sound as receivedfrom the electronic device of the first user.

Consider an example in which the electronic device of the second userreceives a voice message in binaural sound. The second user clicks oractivates the sound to play but is unaware that the sound was sent asbinaural sound. The second user, however, previously set sound settingsin the electronic device to the following: “Request authorization beforeplaying binaural sound.” When the second user clicks or activates thesound, a message displays to the second user: “Play sound as 3D sound?”If the second user accepts, then the sound plays as binaural sound. Ifthe second user denies, then the sound plays as mono sound.

The user or entity desiring to play sound as 3D sound or binaural sounddoes not have privileges, authentication, credentials, or permissions toplay 3D sound or binaural sound to the second user. For example, theuser or entity did not provide a correct access code or password. Asanother example, the user was not attempting access from a mobilemessaging software application that is required to play binaural soundto other users also executing and communicating via the mobile messagingsoftware application.

Consider an example in which two users (a first user and a second user)electronically communicate via a mobile messaging software applicationthat provides voice messaging, calls, texting, and sending graphicalrepresentations with sound (such as emoji and animoji). When theelectronic communication commences, the electronic device of the firstuser sends the electronic device of the second user a request forauthorization to play binaural sound to the second user. The secondelectronic device receives this request and consults privacy settings ofthe second user. These privacy settings include a list of entitiesentitled to play binaural sound to the second user. The first user isnot on this list, so the second electronic device responds to therequest with a denial. The first and second users are still able toelectronically communicate via the software application but theelectronic device the plays all sound received from the first electronicdevice as mono sound or stereo sound. The first user or first electronicdevice is not authorized to play binaural sound to the second user.

If the answer to the determination in block 310 is “yes” then flowproceeds to block 330 that states allow the sound to play to the seconduser as the binaural sound.

The user or entity attempting to play binaural sound to the second useris authorized to do so. For example, this entity has the requisiteprivileges, authentication, credentials, or permissions to play binauralsound to the second user and/or at the electronic device of the seconduser. For example, the user or entity provides a correct access code orpassword. As another example, the user attempting to play binaural soundis doing so from a mobile messaging software application and is allowedsince this user appears as friend or contact with the user to whom thebinaural sound will be played.

Consider an example in which two users (a first user and a second user)electronically communicate via a mobile messaging software applicationthat provides voice messaging in binaural sound, calls in binauralsound, and sending graphical representations with binaural sound (suchas emoji and animoji). When the electronic communication commences, theelectronic device of the first user sends the electronic device of thesecond user a request to play binaural sound. The electronic device ofthe second user displays a message to the second user with this request.For example, the requests displays as follows: “Allow playing of 3Dsound?” The second user affirms the request. In response to thisconfirmation, the electronic device of the first user sends theelectronic device of the second user sounds convolved into binauralsound. The electronic device of the second user plays these sounds inbinaural sound to the second user such that the sounds externallylocalize in empty space or space occupied with a virtual reality (VR) oraugmented reality (AR) image about one meter away from a head of thesecond user.

Consider another example in which two users (a first user and a seconduser) electronically communicate via the mobile messaging softwareapplication. The first user telephones the second user. The electronicdevice of the second user compares the incoming telephone number of thefirst user with a list of entities entitled to play binaural sound tothe second user. This list includes the telephone number as beingauthorized. The electronic device of the second user accepts the call,and the voice of the first user plays as binaural sound to the seconduser.

Consider an example in which a first user records a voice message(“Hello”) with a single microphone in his smartphone and sends thismessage with a talking 3D emoji to the second user. This emoji appearson the display of the smartphone of the second user. When the electronicdevice of the second user receives the sound, the sound is not yetconvolved into binaural sound. The second user activates the emoji toplay the sound. In response to this activation, the electronic device ofthe second user consults a lookup table, determines that the first useris authorized to play binaural sound to the second user, and convolvesthe sound with a DSP in the second electronic device. The second user iswearing headphones and hears the voice of the first user saying “Hello”that externally localizes in empty space next to the second user.

Example embodiments also include the user interacting with one or moreuser interfaces or electronic devices to make the selection per blocks210 and 310. By way of example, the electronic device senses, detects,or receives one or more of a voice command, a hand gesture, body motion,eye movement, head movement, click, menu selection, touch (e.g., on atouch screen), etc. The user can also make the selection with a wand,electronic pen, or other handheld electronic device.

Consider an example in which the first user makes the selection byproviding an instruction or command to an electronic device, such as ahandheld portable electronic device (HPED), wearable electronic device(WED), headphones, a smartphone, an electronic watch, a television ordisplay, a smart speaker, a server, or another electronic device.

Example embodiments enable a user to control who or what has access,authorization, privileges, permissions, credentials, or rights to HRTFsof a user and/or to playing binaural sound or 3D sound to the user. Thisinformation can be established by the user, an electronic device, and/ora software program and stored in memory in various formats andlocations. FIGS. 4 and 5 provide an example of establishing thisinformation via user preferences or privacy settings; though one ofordinary skill in the art will appreciate numerous other ways exists aswell.

Privacy settings or user preferences enable a user or software programto customize, configure, and/or control access to HRTFs and playing ofbinaural sound to the user. This information is stored and subsequentlyretrieved or consulted to enable executing of software programs inaccordance with the configurations of the privacy settings or userpreferences.

FIG. 4 is an electronic device 400 with a display 410 displaying privacysettings 420 that enable a user to control access to HRTFs of the userin accordance with an example embodiment.

By way of example, each of the privacy settings includes a square box toindicate selection of this privacy setting by the user or softwareprogram. Examples of such privacy settings (or their functionalequivalent) include, but are not limited to, the following:

(1) Allow public access to HRTFs for 3D sound. This option enablesmembers of the public or all entities to access the HRTFs of the user.

(2) Allow only friends access to HRTFs for 3D sound. This option limitsaccess to the HRTFs to friends. By way of example, friends include, butare not limited to, entities that the user accepted (e.g., accepted afriend request); relatives, acquaintances, friends, colleagues, orco-workers of the user; people whom the user indicated with a like;people whom the user indicated or designated with access; and peopleextracted from a contact list (e.g., from a contact list of friends in amobile messaging software application).

(3) Deny everyone access to HRTFs for 3D sound. This option is mostrestrictive in that no entity (other than the user himself or herself)has access to the HRTFs.

(4) Notify me when my HRTFs for 3D sound are requested. This optionsgives the user the ability to select in real-time whether an entity doesor does not have access to the HRTFs. For example, a display or voiceinquiry prompts the user to allow or deny a pending user access to theHRTFs.

(5) Allow users in list access to HRTFs for 3D sound. This optionenables the user or software program to add and subtract entities whohave access or do not have access to the HRTFs. In this way, the usercan have a customized list of such entities. Such entities can beentered, stored, and/or identified by name, address, phone number,network or computer address, URL, website, software program, uniqueidentification, etc.

FIG. 5 is an electronic device 500 with a display 510 displaying privacysettings 520 that enable a user to control playing of binaural sound or3D sound to the user in accordance with an example embodiment.

By way of example, each of the privacy settings includes a square box toindicate selection of this privacy setting by the user or softwareprogram. Examples of such privacy settings (or their functionalequivalent) include, but are not limited to, the following:

(1) Allow public to play 3D sound. This option enables members of thepublic or all entities to play 3D sound or binaural sound to the user.

(2) Allow only friends to play 3D sound. This option limits friends toplaying 3D sound to the user. By way of example, friends include, butare not limited to, entities that the user accepted (e.g., accepted afriend request); relatives, acquaintances, friends, colleagues, orco-workers of the user; people whom the user indicated with a like;people whom the user indicated or designated with access; and peopleextracted from a contact list (e.g., from a contact list of friends in amobile messaging software application).

(3) Deny everyone playing of 3D sound. This option is most restrictivein that no entity (other than the user himself or herself) has rights toplay 3D sound to the user.

(4) Notify me before playing 3D sound. This options gives the user theability to select in real-time whether an entity can or cannot play 3Dsound to the user. For example, a display or voice inquiry prompts theuser to allow or deny playing of 3D sound.

(5) Allow users in list access to play 3D sound. This option enables theuser or software program to add and subtract entities who have areauthorized to play or not play 3D sound to the user. In this way, theuser can have a customized list of such entities. Such entities can beentered, stored, and/or identified by name, address, phone number,network or computer address, URL, website, software program, uniqueidentification, etc.

Example embodiments include the first user sending the second user agraphical representation with sound (e.g., sound that is in binauralsound or will play in binaural sound to the second user). The graphicalrepresentation and the sound transmit to the electronic device of thesecond user or an electronic device in communication with the electronicdevice of the second user. For example, the graphical representation andsound transmit from an electronic device of the first user to anelectronic device of the second user or from a server in communicationwith the electronic device of the second user.

Consider an example in which the first user has a HPED or WED thatincludes a wireless transmitter/receiver that wirelessly transits thegraphical representation and/or sound over one or more wireless networksto a HPED or WED of the second user.

Consider an example in which the first and second users communicate witheach other via electronic devices (such as smartphones) that execute amobile messaging software application. The electronic device of firstuser includes a microphone and camera that records a voice and facialimages of the first user that will be played to the second user. A DSPin the first electronic device convolves the sound into binaural sound,processes and stores the facial images as a three-dimensional (3D)graphical representation, and transmits the graphical representation andbinaural sound over a wireless network (e.g., the internet or cellularnetwork) to the electronic device of the second user.

The electronic device also provides or displays the graphicalrepresentation to the second user. For example, the electronic device ofthe second user displays the graphical representation on, thru, or witha display.

Consider an example in which the sound corresponds to or is associatedwith the graphical representation. When the second user, electronicdevice, or software application activates the graphical representation,the binaural sound plays to the second user.

Consider an example in which the first and second user communicate witheach via electronic devices. The electronic device of the first userdisplays a graphical representations (e.g., a plurality of differentemoji or animoji) that play voice greetings or voice messages inbinaural sound. For example, when activated these graphicalrepresentations play a recorded voice that says “Hello” or “How areyou?” or “Hahahaha” or another message. The first user selects agraphical representation that includes a red-shaped heart and soundrecording that plays “I love you.” The first user selects this recordingto play as binaural sound to the second user at a SLP having sphericalcoordinates of (r=1.0 m, θ=30°, ϕ, =0°). The first user selects thegraphical representation, and in response to this selection, thegraphical representation of the red-shape heart appears on a display ofan electronic device of the second user. When the second user selectsthe red-shaped heart a voice says “I love you” to the second user. Thisvoice externally localizes to the second user as binaural sound andoriginates in empty space away from the head of the second user at theSLP of (r=1.0 m, θ=30°, =0°) selected by the first user.

Consider an example in which an electronic device of the first userdisplays a 3D emoji. When the first user selects this emoji, microphonesrecord his or her voice message as a voice message that will be playedto the second user. Selection of this emoji automatically designates thevoice message as being in binaural sound. Further, selection of thisemoji automatically applies a SLP where the sound will localize to thesecond user. For example, this location is set as a default location orpreviously set by the first user. A DSP in the first electronic deviceor a server in communication with this electronic device convolves thesound into binaural sound. The first user sends the 3D emoji to thesecond user. When the 3D emoji appears on the display of the electronicdevice of the second user, the 3D emoji automatically plays the voicemessage in binaural sound to the second user. The voice messageexternally localizes to the default location or previously set location(e.g., one meter away from the head of the second user and to a leftside or right side of the face of the second user).

When sound is selected to be binaural sound, then the sound plays andexternally localizes outside the head of the second user. For example,the sound originates or emanates from a SLP that is in empty space(e.g., where no physical or tangible object exists) or in occupiedspace.

Consider an example in which the first electronic device receives orobtains one or more head-related transfer functions (HRTFs) forconvolving sound to the second user. These HRTFs include generic HRTFsor customized HRTFs to the second user. In this way, a processor, suchas a digital signal processor (DSP), in the first electronic device orin communication with the first electronic device (e.g., a server)processes or convolves the sound before the sound is transmitted and/orprovided to the electronic device of the second user. When the secondelectronic device receives or obtains the sound, the sound is alreadyprocessed or convolved and ready to play as binaural sound. A processorin the second electronic device is not required to process or convolvethe sound into binaural sound so the sound can immediately play to thesecond user without consuming processing resources of the secondelectronic device or without requiring processing time before beingplayed to the second user.

Example embodiments include expediting playing or processing of thesound. One embodiment, for example, expedites processing or convolvingof the sound of the graphical representation at the second electronicdevice by convolving, with a digital signal processor (DSP), the soundwith HRTFs to change the sound into the binaural sound beforetransmitting the graphical representation and the sound from the firstelectronic device to the second electronic device and in response toreceiving the selection of the first visual indication at the firstelectronic device. The DSP can be located in the first electronic deviceor a server in communication with the first and/or second electronicdevices. When the electronic device of the second user receives thegraphical representation and/or sound, the sound is already convolvedinto binaural sound as selected by the first user and is ready forimmediate play to the second user. The second user does not have to waitfor a processor to convolve the sound since this convolution alreadyoccurred. The DSP convolves the sound with the HRTFs before one or moreof (1) the sound is provided to the electronic device of the seconduser, (2) the sound is retrieved by the electronic device of the seconduser, (3) the sound is transmitted to the electronic device of thesecond user (e.g. wirelessly transmitted by a server or the firstelectronic device to the second electronic device), (4) the second userrequests the sound to play, and (5) the second user activates thegraphical representation to play the sound.

Example embodiments include saving processing resources of the DSP inthe electronic device of the second user. One embodiment, for example,processes the sound before the sound is transmitted to, provide to, orretrieved by the second electronic device of the second user. In thisway, the DSP in the second electronic device is not required to devoteprocessing resources to convolution of the sound since the sound isalready convolved into binaural sound and ready for immediate play toexternally localize to the selected SLP.

A graphical representation can include or be associated with sound. Forexample, sound plays to the user when the user, an electronic device, ora software program activates the graphical representation or the soundassociated with the graphical representation.

Consider an example in which two users execute a mobile messagingsoftware application. The first user sends the second user an animatedemoji (or animoji) that when activated or executed says “Hello” to thesecond user.

By way of example, sound can localize to the listener as mono sound orstereo sound when the sound is not convolved and played to the listenerwith headphones, earphones, etc. Mono sound and stereo sound can alsoexternally localize to speakers, such as speakers in a smartphone,stereo speakers in a room, etc. Alternatively, sound externallylocalizes to the listener when the sound is convolved into or capturedas binaural sound or 3D sound. Binaural sound externally localizesoutside or away from the head of the listener and is not required tolocalize to a physical object, such as a speaker. For instance, binauralsound can externally localize one or more meters away from a person at alocation in empty space (e.g., where no speaker exists or no physical ortangible object exists). Binaural sound can also localize to physicalobjects that do not have an electronic speaker, such as localizing to awall or a chair. Sound can also localize as a mix of binaural, mono, andstereo sounds, such as sound commencing as binaural sound thentransitioning to stereo sound to the listener.

If the sound is mono sound or stereo sound and not subsequentlyconvolved with HRTFs or other sound localization information (SLI), thenthe sound will not externally localize as binaural sound. For instance,a user receives a graphical representation and sound recorded in mono orstereo sound. The sound is intended to play to the user as binauralsound, but the electronic device providing the sound does not have HRTFsof the user in order to convolve the sound into binaural sound.

As another example, a user receives a graphical representation and soundcaptured or recorded in binaural sound. The user is wearing headphonesand can hear the sound as binaural sound. The electronic device of theuser, however, is set to only play sound as mono or stereo sound. Forinstance, a permission for binaural sound is set to off, or the user hasnot authorized binaural sound to play with the current executingsoftware application.

As another example, a user is at a location that restricts or prohibitsthe use of binaural sound. For instance, the user is located in aschool, library, government building, or airplane that prohibitslisteners from communicating with binaural sound.

In an example embodiment, a sound file, sound clip, streaming sound, arecording, or other type of sound associates with or corresponds to agraphical representation. Binaural sound plays to the listener when thegraphical representation activates.

In an example embodiment, a user, a listener, a program or softwareapplication, or an electronic device activates the graphicalrepresentation and/or causes the binaural sound to play to the listener.

For example, the listener interacts with a user interface and provides acommand or instruction to play the sound upon receiving the graphicalrepresentation. For instance, the first user performs one or moreactions that include, but are not limited to, clicking or activating anicon, emoji, graphical representation, or other indicia that representsa sound clip, sound file, streaming sound, or recording, selecting thesound from a menu (such as a dropdown menu), selecting the sound from afolder or file (such as a folder or file being displayed to the firstuser), providing a body gesture (such as a hand gesture or hand movementindicating a desire to play the sound), providing head movement or eyemovement (such as the listener moving his or her head in a certaindirection or pattern to indicate selection of the sound), providing avoice command (such as the listener speaking an instruction at a naturallanguage user interface), or taking another action to have the soundplayed to the listener.

As another example, the sound automatically plays. For instance, thesound plays when the listener receives the graphical representation,opens the software program providing the graphical representation, orviews the graphical representation on a display.

As another example, the sound plays when a sender of the sound (e.g.,another user in an electronic communication with the listener) activatesthe sound or designates when the sound plays.

Binaural sound is provided to the listener through one or moreelectronic devices including, but not limited to, one or more ofheadphones, earphones, earbuds, bone conduction devices, or otherelectronic devices with speakers at, in, or near the ears of thelistener. Binaural sound can be processed for crosstalk cancellation andprovided through speakers separate or away from the listener (e.g.,dipole stereo speakers). Electronic devices in communication withheadphones, earphones, and earbuds can provide binaural sound to thelistener (e.g., a smartphone in wireless communication with earphones).

Various types of electronic devices can include or be in communicationwith speakers to provide binaural sound to listeners. Examples of theseelectronic devices include, but are not limited to, wearable electronicglasses, smartphones, head mounted displays (HMDs), optical head mounteddisplays (OHMDs), wearable electronic devices (WEDs), portableelectronic devices (PEDs), handheld portable electronic devices (HPEDs),laptop computers, tablet computers, desktop computers, and otherelectronic devices.

From the point-of-view of the listener, the sound originates or emanatesfrom an object, point, area, or direction. This location for the originof the sound is the sound localization point (SLP). By way of example,the SLP can be an actual point in space (e.g., an empty point in space1-2 meters away from the head of the listener) or a point on or at aphysical or virtual object (e.g., a mouth or head of an augmentedreality (AR) or virtual reality (VR) image). The SLP does not have to beso precise since humans are not always able to localize sound to aparticle point. As such, the SLP can also be a specific or general area(e.g., a location next to and on the right side of the listener) or aspecific or general direction from where the sound originates to thelistener (e.g., a location several meters behind the listener).

When binaural sound is provided to the listener, the listener will hearthe sound as if it originates from the sound source, the source ofsound, or the SLP. The sound, however, does not originate from the soundsource since the sound source or SLP may be an inanimate object with noelectronics or an animate object with no electronics. Alternatively, thesound source or SLP has electronics but does not have the capability togenerate sound (e.g., the sound source has no speakers or sound system).As yet another example, the sound source or SLP has speakers and theability to provide sound but is not providing sound to the listener. Ineach of these examples, the listener perceives the sound to originatefrom the sound source or SLP, but the sound source or SLP does notproduce the sound. Instead, the sound is altered or convolved andprovided to the listener so the sound appears to originate from thesound source or SLP.

In an example embodiment, at least a portion of the sound associatedwith, corresponding to, or provided from the graphical representationexternally localizes away from the head of the listener in empty space(e.g., where no physical or tangible object exists) or occupied space.For example, the sound externally localizes proximate or near thelistener, such as localizing within a few meters of the listener. Forinstance, the SLP where the listener localizes the sound is stationaryor fixed in space (e.g., fixed in space with respect to the user, fixedin space with respect to an object in a room, fixed in space withrespect to an electronic device, fixed in space with respect to anotherobject or person).

By way of example, the SLP can be an actual point in space (e.g., anempty point in space 1-2 meters away from the head of the listener) or apoint on a physical or virtual object (e.g., a mouth or head of anaugmented reality (AR) or virtual reality (VR) image). The SLP does nothave to be so precise since humans are not always able to localize soundto a particle point. As such, the SLP can also be a general area (e.g.,a location next to and on the right side of the listener) or a generaldirection from where the sound originates to the listener (e.g., alocation several meters behind the listener).

Consider an example in which the graphical representation is an emojithat includes a talking animated animal head or human head. When alistener clicks on or activates the emoji, the head talks and thelistener hears the voice as binaural sound that externally localizesabout one meter away from the listener. For instance, the voice isconvolved with head-related transfer functions (HRTFs) having sphericalcoordinates (distance r=1.0 m, elevation ϕ=0°, azimuth θ=30°). Thelistener activates the emoji and hears the voice originate fromspherical coordinates (1.0, 0°, 30°).

By way of example, a computer or electronic device generates the sound(computer-generated sound), or microphones capture and record the soundto be sent. For example, one or more microphones capture the sound asmono sound or stereo sound when the first user speaks a message to thesecond user. As another example, the first electronic device or a serverin communication with the first electronic device includes a pluralitypre-recorded or previously generated sounds.

The sound can be stored in memory of an electronic device, obtained frommemory of an electronic device (such as a computer or server), and/ortransmitted or streamed over one or more networks.

Consider an example in which the first electronic device executes amobile software messaging application that includes hundreds orthousands of sound clips or sound files. The first electronic deviceobtains or has access to these sound clips or sound files and can sendthem to other users of the mobile software messaging application.

Consider an example in which the first electronic device obtains thesound when the first user speaks into microphones in the firstelectronic device or in communication with the first electronic device.The microphones records the voice of the first user as he or she recordsa message or sound to be played to the second user.

The first electronic device transmits the sound and a graphicalrepresentation associated with or corresponding to the sound over one ormore wired or wireless networks (e.g., a cellular network, the internet,etc.). For example, the first electronic device includes a wirelesstransmitter/receiver that sends the sound and graphical representation.

Consider an example in which the first user commands or instructs thesound clip to play to the second user during an electronic communicationbetween the first and second users. In response to this command orinstruction, the first electronic device transmits the sound clip and a3D moving emoji to the second electronic device.

In another example embodiment, a server or another electronic devicetransmits the sound and/or graphical representation to the secondelectronic device. Consider an example in which the first and secondusers talk or message each other with a mobile messaging softwareapplication. The application executes on the electronic devices and oneor more servers. When the first user clicks on a 3D sound emoji, thisaction causes one of the servers to transmit the 3D emoji and sound tothe second electronic device.

The second electronic device receives the sound and the graphicalrepresentation from the first electronic device or another electronicdevice (e.g., a server) in communication with the first electronic. Forexample, the second electronic device includes a wirelesstransmitter/receiver that receives the sound and graphicalrepresentation over one or more networks.

A processor or sound hardware processes or convolves the sound withhead-related transfer functions (HRTFs) or other SLI so the sound willexternally localize as binaural sound to the listener.

Graphical representations can have many sizes, shapes, and forms (e.g.,people, faces, characters, animals, objects, 2D, 3D, etc.). Further, thegraphical representations can be static, such as a 2D or 3D emoji thatdo not move or change facial expressions. Alternatively, the graphicalrepresentations can be dynamic, such as 2D or 3D emoji that move, talk,change facial expressions, rotate, etc. Further yet, graphicalrepresentations in accordance with example embodiments can be presentedas AR images and VR images.

The graphical representations can include or be associated with sound,such as a sound clip, a sound file, a recorded voice message, streamingaudio, etc. The sound can play for a short period of time (e.g., lessthan one second, one second, two seconds, etc.). For example, the soundis a voice saying “Hello” or “Hahahaha” or “Thank you” or another shortaudio message. As another example, the sound is a computer-generated“Beep” or phone ringing or explosion sound. The sound can play forlonger periods of time (e.g., ten seconds, thirty seconds, one minute,several minutes, etc.). For example, the sound is a recorded messagefrom a user during an electronic communication between two users.

By way of example, the sound plays when the listener activates thegraphical representation or another action occurs that initiates oractivates playing of the sound. For example, a first user sends a seconduser an emoji shaped like a heart. This heart appears on a display of anelectronic device of the second user. When the second user clicks on theheart, a voice in binaural sound says “I love you” to the second user.

When sound is already convolved into binaural sound, this sound can beconverted back into mono or stereo sound or played as mono or stereosound. For example, the electronic device plays the sound through asingle speaker. As another example, the electronic device plays the samechannel through both speakers (e.g., play the left channel sound to boththe left and right speakers of the headphones or play the right channelsound to both the left and right speakers of the headphones). As anotherexample, the sound is filtered through cross-talk canceling filters.Filters, for example, can eliminate crosstalk and the HRTFs (e.g., byutilizing an inverse filter, such as a Nelson/Kirkeby inverse filter).

Consider an example embodiment in which a first user and a second usercommunicate with each via their respective electronic devices. Theelectronic device of the second user receives a request for the HRTFsthat are customized to the second user. In response to this request, theelectronic device of the second user transmits the customized HRTFs tothe electronic device of the first user. The electronic device of thefirst user then convolves the sound for the electronic device of thesecond user.

Consider an example embodiment in which a first user and a second usercommunicate with each via their respective electronic devices. Theelectronic device of the second user determines that the first user is afriend of the second user. In response to this determination, theelectronic device of the second user authorizes the electronic device ofthe first user to have access to customized HRTFs of the second user.

Consider an example embodiment in which a first user and a second usercommunicate with each via their respective electronic devices. Theelectronic device of the second user consults privacy settings todetermine whether the first user is authorized to have access tocustomized HRTFs of the second user. In response to consulting theseprivacy settings, the electronic device of the second user denies orallows access to the customized HRTFs. For example, the electronicdevice denies access to the customized HRTFs of the second user when theprivacy settings indicate that the first user is not authorized to haveaccess to the customized HRTFs of the second user. For example, theelectronic device allows access to the customized HRTFs of the seconduser when the privacy settings indicate that the first user isauthorized to have access to the customized HRTFs of the second user.

Consider an example embodiment in which a first user and a second usercommunicate with each via their respective electronic devices. Adetermination is made as to whether the first user has authorization toaccess to customized HRTFs of the second user. The electronic device ofthe first user is providing with the customized H RTFs when the firstuser has the authorization to access the customized HRTFs of the seconduser. The electronic device of the first user is providing with genericHRTFs when the first user does not have the authorization to access thecustomized HRTFs of the second user.

Consider an example embodiment in which a first user and a second usercommunicate with each via their respective electronic devices. Adetermination is made as to whether the first user is authorized toprovide sound to the second user in the binaural sound. The voice of thefirst user plays to the second user in the binaural sound when the firstuser is authorized to provide the sound to the second user in thebinaural sound. The voice of the first user is changed from being in thebinaural sound to being in one of mono sound and stereo sound when thefirst user is not authorized to provide the sound to the second user inthe binaural sound.

Consider an example embodiment in which a first user and a second usercommunicate with each via their respective electronic devices. Duringthis communication (which includes a telephone call), processingresources of a DSP in the electronic device of the second are saved byconvolving the voice of the first user with the DSP in the electronicdevice of the first user before transmitting the voice of the first userto the electronic device of the second user. The voice of the first userin the binaural sound externally localizes to the second user in emptyspace outside of a head of the second user.

Consider an example embodiment in which a first user and a second usercommunicate with each via their respective electronic devices. Adetermination is made as to whether the first user is authorized to sendhis or her voice to the second user in the binaural sound. The voice ofthe first user changes from being provided to the second user inbinaural sound to being provided to the second user in one of mono soundand stereo sound upon determining that the first user is not authorizedto send the voice to the second user in the binaural sound.

Consider an example embodiment in which a first user and a second usercommunicate with each via their respective electronic devices. Theelectronic devices expedite playing of the voice of the first user tothe second user at the electronic device of the second user byconvolving the voice of the first user with the DSP in the electronicdevice of the first user before transmitting the voice of the first userto the electronic device of the second user. The voice of the first userin the binaural sound externally localizes to the second user in emptyspace outside of a head of the second user.

FIG. 6 is an example computer system 600 in accordance with an exampleembodiment.

The computer system 600 includes one or more of a server 610, a database620, a database 624, an electronic device 630, and an electronic device640 in communication over one or more networks 650. User 639 is with oruses electronic device 630, and user 649 is with or uses electronicdevice 640. For illustration, a single server 610, two databases 620 and624, two electronic devices 630 and 640, and two users 639 and 649 areshown, but example embodiments can include a plurality of servers,databases, electronic devices, and users.

Server 610 includes a memory 612 and a processing unit 614. The server610 couples to or communicates with the database 620 that includes HRTFsand other sound localization information 622 and database 624 thatincludes graphical representations (reps) 626 and sound clips 628.

Electronic device 630 includes a processing unit 632 and memory 634 withan HRTF authorizer 636 and a 3D sound authorizer or binaural soundauthorizer 638.

Electronic device 640 includes a processing unit 642 and memory 644 withan HRTF authorizer 646 and a 3D sound authorizer or binaural soundauthorizer 648.

FIG. 7 is an example of an electronic device 700 in accordance with anexample embodiment.

The electronic device 700 includes a processor or processing unit 710,memory 720 with sound clips 722 and graphical representations orgraphical reps 724, a display 730, one or more interfaces 740, awireless transmitter/receiver 750, speakers 760, one or more microphones770, head tracking 780 (such as one or more of an inertial sensor,accelerometer, gyroscope, and magnetometer), and an HRTF authorizerand/or 3D sound authorizer 790.

The HRTF authorizer and the 3D sound authorizer or binaural soundauthorizer include software, hardware, and/or program instructionsdiscussed in figures herein that determine access to HRTFs of users andplaying of 3D sound or binaural sound to users as discussed herein. Forexample, the HRTF authorizer executes one or more blocks associated withFIG. 2, FIG. 4, and other example embodiments discussed herein. Forexample, the 3D sound authorizer executes one or more blocks associatedwith FIG. 3, FIG. 5, and other example embodiments discussed herein.

Memory includes computer readable medium (CRM). Examples of an interfaceinclude, but are not limited to, a network interface, a graphical userinterface, a natural language user interface, a natural user interface,a phone control interface, a reality user interface, a kinetic userinterface, a touchless user interface, an augmented reality userinterface, and/or an interface that combines reality and virtuality.

Sound clips include sound files, sounds, recorded messages (such asvoice messages or other recorded sound), computer-generated sounds, andother sound discussed herein. For example, users can record, exchange,and/or transmit sound clips or sounds. These sound include sendingstreaming sounds or sounds in real-time during an electroniccommunication.

The processor or processing unit includes a processor and/or a digitalsignal processor (DSP). For example, the processing unit includes one ormore of a central processing unit, CPU, digital signal processor (DSP),microprocessor, microcontrollers, field programmable gate arrays (FPGA),application-specific integrated circuits (ASIC), etc. for controllingthe overall operation of memory (such as random access memory (RAM) fortemporary data storage, read only memory (ROM) for permanent datastorage, and firmware).

Consider an example embodiment in which the processing unit includesboth a processor and DSP that communicate with each other and memory andperform operations and tasks that implement one or more blocks of theflow diagram discussed herein. The memory, for example, storesapplications, data, programs, sound clips, algorithms (includingsoftware to implement or assist in implementing example embodiments) andother data.

For example, a processor or DSP executes a convolving process with theretrieved HRTFs or HRIRs (or other transfer functions or impulseresponses) to process sound clips so that the sound is adjusted, placed,or localized for a listener away from but proximate to the head of thelistener. For example, the DSP converts mono or stereo sound to binauralsound so this binaural sound externally localizes to the user. The DSPcan also receive binaural sound and move its localization point, add orremove impulse responses (such as RIRs), and perform other functions.

For example, an electronic device or software program convolves and/orprocesses the sound captured at the microphones of an electronic deviceand provides this convolved sound to the listener so the listener canlocalize the sound and hear it. The listener can experience a resultinglocalization externally (such as at a sound localization point (SLP)associated with near field HRTFs and far field HRTFs) or internally(such as monaural sound or stereo sound).

The memory stores HRTFs, HRIRs, BRTFs, BRIRs, RTFs, RIRs, or othertransfer functions and/or impulse responses for processing and/orconvolving sound. The memory can also store instructions for executingone or more example embodiments. Further, the memory can store thesound, graphical representations, and other information and instructionsdiscussed herein.

The electronic device provides sound to the users through one or morespeakers. Alternatively or in addition to the speakers, the electronicdevice can communicate with headphones, earphones, earbuds, boneconduction devices, or another electronic device that provides sound tothe user.

The networks include one or more of a cellular network, a public switchtelephone network, the Internet, a local area network (LAN), a wide areanetwork (WAN), a metropolitan area network (MAN), a personal areanetwork (PAN), home area network (HAM), and other public and/or privatenetworks. Additionally, the electronic devices need not communicate witheach other through a network. As one example, electronic devices coupletogether via one or more wires, such as a direct wired-connection. Asanother example, electronic devices communicate directly through awireless protocol, such as Bluetooth, near field communication (NFC), orother wireless communication protocol.

By way of example, a computer and an electronic device include, but arenot limited to, handheld portable electronic devices (HPEDs), wearableelectronic glasses, electronic or smart watches, wearable electronicdevices (WEDs), smart earphones or hearables, electronic devices withcellular or mobile phone capabilities or subscriber identificationmodule (SIM) cards, desktop computers, servers, portable computers (suchas tablet and notebook computers), smartphones, head mounted displays(HMDs), optical head mounted displays (OHMDs), headphones, and otherelectronic devices with a processor or processing unit, a memory, a DSP.

Example embodiments are not limited to HRTFs but also include othersound transfer functions and sound impulse responses including, but notlimited to, head related impulse responses (HRIRs), room transferfunctions (RTFs), room impulse responses (RIRs), binaural room impulseresponses (BRIRs), binaural room transfer functions (BRTFs), headphonetransfer functions (HPTFs), etc.

Example embodiments can be executed with one or more integrated circuitsthat are specifically customized, designed, or configured to execute oneor more blocks discussed herein. For example, the electronic devicesinclude a specialized or custom processor or microprocessor orsemiconductor intellectual property (SIP) core or digital signalprocessor (DSP) with a hardware architecture optimized for convolvingsound and executing one or more example embodiments.

Consider an example in which the HPED (including headphones) includes acustomized or dedicated DSP that executes one or more blocks discussedherein (including processing and/or convolving sound into binaural soundfor sound clips). Such a DSP has a better power performance or powerefficiency compared to a general-purpose microprocessor and is moresuitable for a HPED or WED due to power consumption constraints of theHPED or WED. The DSP can also include a specialized hardwarearchitecture, such as a special or specialized memory architecture tosimultaneously fetch or pre-fetch multiple data and/or instructionsconcurrently to increase execution speed and sound processing efficiencyand to quickly correct errors while sound externally localizes to theuser. By way of example, streaming sound data (such as sound data in atelephone call or software game application) is processed and convolvedwith a specialized memory architecture (such as the Harvard architectureor the Modified von Neumann architecture). The DSP can also provide alower-cost solution compared to a general-purpose microprocessor thatexecutes digital signal processing and convolving algorithms. The DSPcan also provide functions as an application processor ormicrocontroller. The DSP can also prefetch sound clips and other soundfrom memory to expedite convolution.

Consider an example in which a customized DSP includes one or morespecial instruction sets for multiply-accumulate operations (MACoperations), such as convolving with transfer functions and/or impulseresponses (such as HRTFs, HRIRs, BRIRs, et al.), executing Fast FourierTransforms (FFTs), executing finite impulse response (FIR) filtering,and executing instructions to increase parallelism.

Consider another example in which sound clips, graphicalrepresentations, and/or HRTFs (or other transfer functions or impulseresponses) are stored or cached in the DSP memory or local memoryrelatively close to the DSP to expedite binaural sound processing.

As used herein, “customized HRTFs” or “HRTFs that are customized” arespecific to an anatomy of a particular listener and are based on a sizeand/or shape of the head and/or ears of the listener.

As used herein, an “emoji” is a graphical representation that includesimages, symbols, or icons sent between users in electroniccommunications (such as text messages, e-mail, and social media) toexpress an emotional attitude of the writer, convey information, orcommunicate an message. Emojis can provide sound when activated orexecuted.

As used herein, “empty space” is a location that is not occupied by atangible object.

As used herein, “graphical representations” include, but are not limitedto, emoji, emoticons, animoji, icons, stickers, folders, documents,files, text or words, pictures, images, and other visible indicia thatdisplay on, thru, or with an electronic device. Furthermore, thesegraphical representations can be two-dimensional (2D), three-dimensional(3D), virtual reality (VR) images, augmented reality (AR) images, staticor non-moving, moving, and other types of images.

As used herein, “headphones” or “earphones” include a left and rightover-ear ear cup, on-ear pad, or in-ear monitor (IEM) with one or morespeakers or drivers for a left and a right ear of a wearer. The left andright cup, pad, or IEM may be connected with a band, connector, wire, orhousing, or one or both cups, pads, or IEMs may operate wirelessly beingunconnected to the other. The drivers may rest on, in, or around theears of the wearer, or mounted near the ears without touching the ears.

As used herein, the word “proximate” means near. For example, binauralsound that externally localizes away from but proximate to a userlocalizes within three meters of the head of the user.

As used herein, a “sound localization point” or “SLP” is a locationwhere a listener localizes sound. A SLP can be internal (such asmonaural sound that localizes inside a head of a listener), or a SLP canbe external (such as binaural sound that externally localizes to a pointor an area that is away from but proximate to the person or away frombut not near the person). A SLP can be a single point such as onedefined by a single pair of HRTFs or a SLP can be a zone or shape orvolume or general area. Further, in some instances, multiple impulseresponses or transfer functions can be processed to convolve sounds to aplace within the boundary of the SLP. In some instances, a SLP may nothave access to a particular HRTF necessary to localize sound at the SLPfor a particular user, or a particular HRTF may not have been created. ASLP may not require a HRTF in order to localize sound for a user, suchas an internalized SLP, or a SLP may be rendered by adjusting an ITDand/or ILD or other human audial cues.

As used herein, “sound localization information” or “SLI” is informationthat is used to process or convolve sound so the sound externallylocalizes as binaural sound to a listener.

As used herein, a “telephone call,” or a “electronic call” is aconnection over a wired and/or wireless network between a calling personor user and a called person or user. Telephone calls can use landlines,mobile phones, satellite phones, HPEDs, voice personal assistants(VPAs), computers, and other portable and non-portable electronicdevices. Further, telephone calls can be placed through one or more of apublic switched telephone network, the internet, and various types ofnetworks (such as Wide Area Networks or WANs, Local Area Networks orLANs, Personal Area Networks or PANs, Campus Area Networks or CANs,etc.). Telephone calls include other types of telephony including Voiceover Internet Protocol (VoIP) calls, internet telephone calls, in-gamecalls, telepresence, etc.

As used herein, a “user” or a “listener” is a person (i.e., a humanbeing). These terms can also be a software program (including an IPA orIUA), hardware (such as a processor or processing unit), an electronicdevice or a computer (such as a speaking robot or avatar shaped like ahuman with microphones in its ears or about six inches apart).

In some example embodiments, the methods illustrated herein and data andinstructions associated therewith, are stored in respective storagedevices that are implemented as computer-readable and/ormachine-readable storage media, physical or tangible media, and/ornon-transitory storage media. These storage media include differentforms of memory including semiconductor memory devices such as DRAM, orSRAM, Erasable and Programmable Read-Only Memories (EPROMs),Electrically Erasable and Programmable Read-Only Memories (EEPROMs) andflash memories; magnetic disks such as fixed and removable disks; othermagnetic media including tape; optical media such as Compact Disks (CDs)or Digital Versatile Disks (DVDs). Note that the instructions of thesoftware discussed above can be provided on computer-readable ormachine-readable storage medium, or alternatively, can be provided onmultiple computer-readable or machine-readable storage media distributedin a large system having possibly plural nodes. Such computer-readableor machine-readable medium or media is (are) considered to be part of anarticle (or article of manufacture). An article or article ofmanufacture can refer to a manufactured single component or multiplecomponents.

Blocks and/or methods discussed herein can be executed and/or made by auser, a user agent (including machine learning agents and intelligentuser agents), a software application, an electronic device, a computer,firmware, hardware, a process, a computer system, and/or an intelligentpersonal assistant. Furthermore, blocks and/or methods discussed hereincan be executed automatically with or without instruction from a user.

What is claimed is:
 1. A method executed by one or more electronicdevices that expedites playing of sound during a telephone call betweena first person and a second person, the method comprising: receiving,during the telephone call and at a microphone of a handheld portableelectronic device (HPED) of the first person, a voice of the firstperson; convolving, during the telephone call and with a digital signalprocessor (DSP) in the HPED of the first person, the voice of the firstperson with head-related transfer functions (HRTFs) of the second personto change the voice of the first person into binaural sound;transmitting, during the telephone call and over a wireless network, thevoice of the first person in the binaural sound to a HPED of the secondperson; and playing, during the telephone call and with the HPED of thesecond person, the voice of the first person in the binaural sound thatwas convolved by the DSP in the HPED of the first person such that thevoice externally localizes to the second person in empty space away froma head of the second person.
 2. The method of claim 1 furthercomprising: receiving, at the HPED of the second person and from theHPED of the first person, a request for the HRTFs that are customized tothe second person; and transmitting, over the wireless network and inresponse to the request, the HRTFs that are customized to the secondperson from the HPED of the second person to the HPED of the firstperson.
 3. The method of claim 1 further comprising: determining, withthe HPED of the second person, that the first person is a friend of thesecond person; and authorizing, by the HPED of the second person, theHPED of the first person to have access to the HRTFs of the secondperson in response to determining that the first person is the friend ofthe second person.
 4. The method of claim 1 further comprising:consulting, with the HPED of the second person, privacy settings todetermine whether the first person is authorized to have access tocustomized HRTFs of the second person; denying, in response toconsulting the privacy settings, the HPED of the first person access tothe customized HRTFs of the second person when the privacy settingsindicate that the first person is not authorized to have access to thecustomized HRTFs of the second person; and allowing, in response toconsulting the privacy settings, the HPED of the first person access tothe customized HRTFs of the second person when the privacy settingsindicate that the first person is authorized to have access to thecustomized HRTFs of the second person.
 5. The method of claim 1 furthercomprising: determining whether the first person has authorization toaccess to customized HRTFs of the second person; providing the HPED ofthe first person with the customized HRTFs when the first person has theauthorization to access the customized HRTFs of the second person; andproviding the HPED of the first person with generic HRTFs when the firstdoes not have the authorization to access the customized HRTFs of thesecond person.
 6. The method of claim 1 further comprising: displaying,with a display of the HPED of the second person, privacy settings thatenable the second user to select people other than the second personhave access to customized HRTFs of the second user.
 7. The method ofclaim 1 further comprising: determining, during the telephone call andwith the HPED of the second person, whether the first person isauthorized to provide sound to the second person in the binaural sound;playing, during the telephone call and with the HPED of the secondperson, the voice of the first person in the binaural sound when thefirst person is authorized to provide the sound to the second person inthe binaural sound; and changing, during the telephone call and with theHPED of the second person, the voice of the first person from being inthe binaural sound to being in one of mono sound and stereo sound whenthe first person is not authorized to provide the sound to the secondperson in the binaural sound.
 8. A non-transitory computer-readablestorage medium that one or more electronic devices execute to improve atelephone call between a first person and a second person, the methodcomprising: obtaining, at an electronic device of the first person,head-related transfer functions (HRTFs) that are customized to thesecond person; convolving, during the telephone call and with a digitalsignal processor (DSP) in the electronic device of the first person, avoice of the first person with the HRTFs that are customized to thesecond person; transmitting, during the telephone call and from theelectronic device of the first person to an electronic device of thesecond person, the voice of the first person in binaural sound that wasconvolved by the DSP with the HRTFs that are customized to the secondperson; and playing, during the telephone call and with the electronicdevice of the second person, the voice of the first person in thebinaural sound that was convolved by the DSP with the HRTFs that arecustomized to the second person.
 9. The non-transitory computer-readablestorage medium of claim 8 further comprising: saving, during thetelephone call, processing resources of a DSP in the electronic deviceof the second person by convolving the voice of the first person withthe DSP in the electronic device of the first person before transmittingthe voice of the first person to the electronic device of the secondperson, wherein the voice of the first person in the binaural soundexternally localizes to the second person in empty space outside of ahead of the second person.
 10. The non-transitory computer-readablestorage medium of claim 8 further comprising: determining whether thefirst person is authorized to send the voice to the second person in thebinaural sound; and changing the voice of the first person from beingprovided to the second person in the binaural sound to being provided tothe second person in one of mono sound and stereo sound upon determiningthat the first person is not authorized to send the voice to the secondperson in the binaural sound.
 11. The non-transitory computer-readablestorage medium of claim 8 further comprising: expediting playing of thevoice of the first person to the second person at the electronic deviceof the second person by convolving the voice of the first person withthe DSP in the electronic device of the first person before transmittingthe voice of the first person to the electronic device of the secondperson, wherein the voice of the first person in the binaural soundexternally localizes to the second person in empty space outside of ahead of the second person.
 12. The non-transitory computer-readablestorage medium of claim 8 further comprising: determining whether thesecond person provided authorization to the first person to access theHRTFs that are customized to the second person; allowing access to theHRTFs that are customized to the second person upon determining that thesecond person provided the authorization; and denying access to theHRTFs that are customized to the second person upon determining that thesecond person provided the authorization.
 13. The non-transitorycomputer-readable storage medium of claim 8 further comprising:determining whether the second person provided authorization for thefirst person to play the voice of the first person in the binaural soundto the second person; allowing the electronic device of the secondperson to play the voice of the first person in the binaural sound tothe second person upon determining that the second person provided theauthorization for the first person; and denying the electronic device ofthe second person to play the voice of the first person in the binauralsound to the second person upon determining that the second person didnot provide the authorization for the first person.
 14. Thenon-transitory computer-readable storage medium of claim 8, wherein theelectronic device of the first person is a smartphone.
 15. A methodexecuted by one or more electronic devices during a telephone callbetween a first person with an electronic device and a second personwith an electronic device, the method comprising: expediting, during thetelephone call, processing of a voice of the first person at theelectronic device of the second person by: convolving, during thetelephone call and with a digital signal processor (DSP) in theelectronic device of the first person, the voice of the first personwith head-related transfer functions (HRTFs) that are customized to thesecond person; transmitting, during the telephone call and from theelectronic device of the first person to the electronic device of thesecond person, the voice of the first person already convolved by theDSP in the electronic device of the first person with the HRTFs that arecustomized to the second person; and playing, during the telephone calland at the electronic device of the second person, the voice of thefirst person already convolved by the DSP in the electronic device ofthe first person with the HRTFs that are customized to the secondperson.
 16. A method of claim 15 further comprising: saving processingresources of a DSP in the electronic device of the second person sincethe voice of the first person is already convolved into binaural soundby the DSP in the electronic device of the first person, wherein thevoice of the first person externally localizes to the second person inempty space one meter away from a head of the second person.
 17. Amethod of claim 15 further comprising: determining that the first personis not authorized to play binaural sound to the second person; andchanging the voice of the first person from being played to the secondperson in the binaural sound to being played in one of mono sound andstereo sound in response to determining that the first person is notauthorized to play the binaural sound to the second person.
 18. A methodof claim 15 further comprising: receiving, at the electronic device ofthe second person, a request from the electronic device of the firstperson for access to the HRTFs that are customized to the second person;allowing access to the HRTFs that are customized to the second person inresponse to determining that the first person is authorized to have theHRTFs that are customized to the second person; and denying access tothe HRTFs that are customized to the second person in response todetermining that the first person is not authorized to have the HRTFsthat are customized to the second person.
 19. A method of claim 15further comprising: displaying, with a display of the electronic deviceof the second person, privacy settings that enable the second person toselect which people other than the second person have access to thecustomized HRTFs of the second person.
 20. A method of claim 15 furthercomprising: displaying, with a display of the electronic device of thesecond person, privacy settings that enable the second person to selectwhich people other than the second person have authorized to the playsound in binaural sound to the second person.